Add pool balancing strategy #233
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This change adds the ability to specify the pool balancing strategy to use when processing queued jobs. Before this change, GARM would round-robin through all pools that matched the set of tags requested by queued jobs. When round-robin (default) is used for an entity (repo, org or enterprise) and you have 2 pools defined for that entity with a common set of tags that match 10 jobs (for example), then those jobs would trigger the creation of a new runner in each of the two pools in turn. Job 1 would go to pool 1, job 2 would go to pool 2, job 3 to pool 1, job 4 to pool 2 and so on. When "stack" is used, those same 10 jobs would trigger the creation of a new runner in the pool with the highest priority, every time. In both cases, if a pool is full, the next one would be tried automatically. For the stack case, this would mean that if pool 2 had a priority of 10 and pool 1 would have a priority of 5, pool 2 would be saturated first, then pool 1. Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Without preloading the entity we're listing pools for, we don't get that info when listing pools for a repo/org/enterprise. Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
gabriel-samfira
changed the title
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
gabriel-samfira
changed the title
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
gabriel-samfira
force-pushed
the
add-balancing-strategy
branch
from
5db1a7e to
ac29af6
Compare
Signed-off-by: Gabriel Adrian Samfira <gsamfira@cloudbasesolutions.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change adds the ability to specify the pool balancing strategy to use when processing queued jobs. Before this change, GARM would round-robin through all pools that matched the set of tags requested by queued jobs.
When round-robin (default) is used for an entity (repo, org or enterprise) and you have 2 pools defined for that entity with a common set of tags that match 10 jobs (for example), then those jobs would trigger the creation of a new runner in each of the two pools in turn. Job 1 would go to pool 1, job 2 would go to pool 2, job 3 to pool 1, job 4 to pool 2 and so on.
When "stack" is used, those same 10 jobs would trigger the creation of a new runner in the pool with the highest priority, every time.
In both cases, if a pool is full, the next one would be tried automatically.
For the stack case, this would mean that if pool 2 had a priority of 10 and pool 1 would have a priority of 5, pool 2 would be saturated first, then pool 1.
To use this we would first need to set a priority on the pools:
Now we need to switch the
70227434-e7c0-4db1-8c17-e9ae3683f61erepository tostack:And now, when new jobs come in, the
577627f4-1add-4a45-9c62-3a7cbdec8403should always be preferred, until it is full.Github currently doesn't allow us to prioritize which runners pick up jobs first, but we can at least decide which pools spin up runners first. This should at least offer some relief for issues like the one detailed here: